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PREFACE 

This  document  was  prepared  by  the  Data  Management  Committee  (DMC),  Data  Sciences 
Group  (DSG),  of  the  Range  Commanders  Council  (RCC)  under  Task  DS-03.  The  DMC 
performed  research  and  analysis  of  the  commonalities  in  the  test  and  evaluation  (T&E)  processes 
across  RCC  member  ranges  and  developed  this  document  of  best  practices  and  lessons  learned 
on  metadata  usage  in  range  applications.  The  document  provides  the  best  practices  based  on 
success  stories  and  lessons  learned  at  various  ranges  and  identifies  what  to  avoid  when  initiating 
a  metadata  effort.  The  foundation  is  laid  for  a  metadata  standard  at  all  ranges  which  will  assist  in 
achieving  commonality  and  for  facilitation  of  real-time  and  post-test  common  access 
methodology  for  local  and  distributed  test  events. 

The  RCC  would  like  to  thank  the  Data  Management  Committee  for  the  hard  work  in 
developing  this  document. 

Task  Lead:  Annette  Weisenseel  AFFTC 

Air  Force  Flight  Test  Center  (AFFTC) 

812  TSS/CM 
307  E.  Popson  Ave 

Edwards  Air  Force  Base  (AFB)  CA  93524-6680 
Phone:  (661)277-1240  DSN  527-1240 
Fax:  (661)277-0249  DSN  527-0249 

Email:  annette ,  weisenseel@ed  wards,  af.  mil 


Data  Management  Committee  Members: 


Jason  Kaza 
Dave  Quick 


-  Yuma  Proving  Ground  (YPG) 

-  Naval  Undersea  Warfare  Center  Division  Keyport 
NUWCDIVKPT 


Dave  Salas 
Steve  Powell 
Tracy  Mullendore  - 
John  Hamilton 


White  Sands  Missile  Range  (WSMR) 

Arnold  Engineering  Development  Center  (AEDC) 
Dugway  Proving  Ground  (DPG) 

Knowledge  Based  Systems,  Inc.  (KBSI) 


Please  direct  any  questions  to: 


Secretariat,  Range  Commanders  Council 

ATTN:  TEDT-WS-RCC 

1510  Headquarters  Avenue 

White  Sands  Missile  Range,  NM  88002-5 110 

Phone:  (575)678-1107  DSN  258-1107 

Fax:  (575)678-7519  DSN  258-7519 

Email:  usarmv.wsmr.atec.list.rcc@mail.mil 
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CHAPTER  1 

BACKGROUND 


1.1  Purpose 

Modem  test  and  evaluation  (T&E)  efforts  produce  large  amounts  of  information  that  is 
external  to  the  actual  measurements  being  acquired,  such  as  test  requirements,  descriptions  of  the 
test  article,  data  format  descriptions,  and  much  more.  Maintaining  access  to  and  understanding 
this  metadata  is  crucial  to  understanding  the  data  itself.  This  understanding  becomes  especially 
critical  when  test  data  must  be  revisited  months,  or  even  years,  after  a  test  is  completed.  The 
value  of  the  original  data  is  diminished  without  the  metadata.  The  ability  to  review  all  of  the 
metadata  associated  with  a  previously  perfonned  test  provides  a  complete  picture  of  the 
circumstances  in  which  the  data  was  gathered.  Therefore,  the  analysis  of  the  older  data  will  be 
easier  and  more  effective. 

The  purpose  of  this  document  is  to  describe  “best  practices”  gathered  from  the  T&E 
community  regarding  the  creation,  utilization,  and  storage  of  T&E  metadata.  The  information 
contained  herein  does  not  completely  define  a  general  T&E  process,  nor  does  it  serve  as  a 
mandate  on  test  organizations  to  implement  the  identified  practices.  Instead,  this  document 
merely  suggests  means  by  which  individual  organizations,  and  the  T&E  community  as  a  whole, 
can  improve  the  way  T&E  metadata  is  handled. 

Note:  A  very  useful  companion  document  is  the  Range  Commanders  Council  (RCC) 

Document  176-11,  T&E  Metadata  Reference  Model  document  (Reference  a). 

1.2  Scope 

The  scope  of  this  document  is  “T&E  Metadata”.  The  most  general  definition  of  metadata 
is  “data  about  data.”  For  this  effort,  we  define  “T&E  data”  to  be  the  actual  acquired 
measurements  from  a  test.  In  Reference  b,  T&E  Metadata  is  defined  as  follows: 

“T&E  metadata  is  any  information  that  provides  additional  description  or  context  to  the 
T&E  data.  This  covers  a  broad  spectrum  of  information,  ranging  from  the  initial 
requirements  and  motivation  for  the  test,  to  the  test  article  and  instrumentation 
modifications  required  to  perform  the  test,  to  the  description  of  the  packet  format  in 
which  the  data  is  transported.  ” 

This  definition  includes  requirements,  test  plans,  safety  reports,  instrumentation  hardware 
descriptions,  measurement  lists,  TMATS  files,  and  other  relevant  data.  The  breadth  of  this  effort 
is  the  complete  set  of  T&E  metadata  as  described  above.  In  comparison,  the  depth  is  relatively 
small.  The  best  practices  described  herein  are  meant  to  be  abstract  enough  to  be  relevant  for  all 
T&E  organizations,  regardless  of  the  type  of  article  being  tested,  the  systems  being  used,  or  the 
geographical  location  of  the  test. 
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1.3  Methodology 

The  research  for  and  documentation  of  these  metadata  best  practices  was  assisted  by  a 
Phase  II  Small  Business  Innovative  Research  (SBIR)  Project  funded  by  Edwards  AFB  and 
carried  out  by  Knowledge  Based  Systems,  Inc  (KBSI).  The  project,  named  “T&E  Metadata 
Plaza”  (TEMPL),  seeks  to  develop  a  methodology  and  a  suite  of  tools  to  improve  management  of 
diverse  types  of  T&E  metadata  (Reference  c). 

1.3.1  Actions  Taken  in  Developing  this  Document.  The  following  major  activities  taken 
included  information  gathering,  coordination  with  the  committee,  and  creation  of  the  document. 

a.  Information  Gathering.  In  order  to  gather  the  information  necessary  to  develop  this 
document,  we  visited  a  diverse  set  of  T&E  organizations  across  the  country  and 
discussed  current  metadata  artifacts,  practices,  and  issues  with  the  various  metadata 
developers  and  users  at  each  site.  The  purpose  of  each  visit  was  to  gather  information 
not  only  for  this  document,  but  also  for  the  T&E  Metadata  Reference  Model 
described  in  Reference  a. 

Over  the  course  of  4  months,  members  of  the  KBSI  project  team  visited  seven 
different  test  organizations.  Table  1-1  lists  the  facilities  visited,  the  date  of  each  visit, 
and  the  key  points  of  contact. 


TABLE  1-1.  T&E  ORGANIZATION  VISITS 

Service 

Facility 

POC 

Trip  Date 

KBSI 

Attendees 

Air  Force 

Edwards  AFB 

Charles  Jones 

4/15/2010 

John  Hamilton 
Tim  Darr 

NASA 

NASA  Dryden 

Robert  Harvey 

4/16/2010 

John  Hamilton 
Tim  Darr 

Army 

Yuma 

Proving  Ground 

Jason  Kaza 

6/7/2010 

John  Hamilton 
Byon  Williams 

Commercial 

Boeing 

Lee  Eccles 

6/15/2010 

John  Hamilton 
Byon  Williams 

Anny 

Aberdeen  Proving  Ground 

George  Bartlett 

6/29/2010 

Byon  Williams 

Navy 

Keyport 

David  Quick 

7/13/2010 

John  Hamilton 
Byon  Williams 

Navy 

Patuxent  River 

Eric  Harvey 

7/26/2010 

John  Hamilton 
Byon  Williams 

Anny 

White  Sands  Missile  Range 

Dave  Salas 

8/3/2010 

John  Hamilton 
Byon  Williams 
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During  each  visit,  the  KBSI  members  met  with  several  different  groups  within  the 
organization.  These  groups  were  coordinated  by  the  primary  point  of  contact  for  each 
facility,  but  were  typically  grouped  according  to  their  role  in  the  test  process. 

Each  group  was  asked  to  describe  their  typical  test  process.  During  each  of  these 
process  descriptions,  the  specific  metadata  artifacts  that  were  created,  modified,  or 
used  at  each  step  were  documented,  along  with  how  they  are  used,  where  they  are 
stored,  and  how  they  are  accessed.  Members  were  also  asked  to  share  any  specific 
difficulties  they  have  encountered  in  dealing  with  metadata  or  any  specific  activities 
they  perform  that  make  management  of  T&E  metadata  easier. 

In  preparation  for  each  visit,  the  group  participants  were  sent  a  list  of  questions  to 
consider  in  advance  of  each  visit.  This  list  contained  the  following  questions: 

(1)  What  metadata/document  management  systems  do  you  use,  and  how  well  do  they 
work? 

(2)  What  metadata-related  issues  have  you  solved/improved,  and  how?  What  hasn’t 
worked  and  why? 

(3)  Are  there  any  metadata-related  issues  that  you  would  like  solved? 

(4)  What  is  missing  from  the  current  T&E  Metadata  Reference  Model? 

(5)  Do  you  have  suggestions  for  improving  the  Reference  Model  terminology? 

(6)  What  metadata  standards  do  you  use  today?  Examples  include  the  Institute  of 
Electrical  and  Electronics  Engineers,  IEEE  1484.12.1-2002  Standard;  the 
integrated  Network  Enhanced  Telemetry  (iNET);  the  Motion  Imagery  Standards 
Group;  the  IRIG/RCC  Telemetry  Standards  106  Chapter  9  and  Chapter  10;  and 
the  Joint  Mission  Environment  Test  Capability  (JMETC)  user  groups. 

(7)  Are  you  aware  of  any  other  efforts  that  could  augment  this  effort?  Examples  are 
Small  Business  Innovative  Research  (SBIR)  programs,  Instrumentation  and 
Modernization  (I&M)  programs,  working  groups,  and  so  forth. 

b.  Coordination  Between  KBSI  and  the  Committee.  Throughout  this  effort,  KBSI 
conducted  monthly  teleconferences  with  members  of  the  RCC-DSG  Data 
Management  committee  to  review  the  information  gathered  from  each  trip  and  to 
solicit  feedback.  Additionally,  a  collaboration  website  by  KBSI  was  set  up  where 
committee  members  could  weigh  in  on  the  progress  of  tasks  between  meetings.  The 
collaboration  site  includes  the  following  major  components: 

(1)  A  document  repository  for  storage  of  current  documents,  minutes,  and  records 
from  all  meetings. 

(2)  A  discussion  forum  for  posting  of  questions  and  feedback  regarding  the  tasks  and 
deliverables. 

c.  Development  of  the  Document.  During  the  information-gathering  phase,  a  running 
list  of  metadata  best  practices  was  maintained,  along  with  the  organization(s)  who 
suggested  each  one.  At  each  monthly  telecon  with  the  Data  Management  Committee, 
this  list  was  reviewed,  discussed,  and  modified. 
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Following  the  completion  of  the  various  site  visits,  the  list  of  best  practices  was 
again  reviewed  by  the  committee,  at  which  time  the  various  practices  were  grouped 
into  categories  and  edited  to  ensure  applicability  to  the  T&E  community  as  a  whole. 
These  categories  and  practices  are  documented  in  the  next  section. 
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CHAPTER  2 

BEST  PRACTICES 


2.1  Utilization  of  Standards 

One  of  the  biggest  challenges  facing  T&E  organizations  with  respect  to  metadata 
management  is  the  wide  range  of  proprietary  metadata  fonnats  and  differing  terminologies  used 
by  the  various  organizations.  This  lack  of  standardization  makes  it  difficult  for  one  organization 
to  share  metadata  (and  thus,  data)  with  another  organization.  This  issue  exists  not  only  between 
test  organizations  but  also  within  individual  organizations. 

If  the  metadata  associated  with  a  specific  aspect  of  a  test  is  stored  in  a  proprietary  fonnat, 
then  any  organization  needing  access  to  that  metadata  is  required  to  have  systems  capable  of 
interpreting  that  format.  Without  standards,  the  number  of  fonnats  (and  hence  the  complexity 
and/or  number  of  systems)  dramatically  increases. 

Additionally,  confusion  can  occur  when  metadata  is  shared  between  organizations 
lacking  a  common  terminology.  Without  specifically  defined  terminology,  the  meanings  of 
many  terms  used  to  describe  tests  can  be  ambiguous.  For  example,  one  organization  may 
typically  use  the  term  “transmission”  to  refer  to  vehicle  transmissions,  while  another 
organization  may  use  the  same  term  to  refer  to  the  sending  of  information  via  radio  waves. 

Even  if  metadata  is  not  shared  outside  of  a  particular  organization,  lack  of  standardization  can 
cause  problems.  Proprietary  formats  are  often  tied  to  a  particular  commercial  entity  or  product, 
and  may  not  be  well  documented.  This  makes  it  difficult,  if  not  impossible,  for  an  organization 
to  migrate  to  a  new  system  that  uses  a  different  format.  In  addition,  products  become  obsolete 
over  time,  and  companies  go  out  of  business.  When  a  proprietary  format  is  used  for  the 
archiving  of  metadata,  there  is  a  risk  that  the  information  will  no  longer  be  accessible  beyond  the 
lifetime  of  the  company  or  product. 

These  problems  are  eliminated  by  using  widely  accepted,  well-documented  standards. 
Given  the  broad  scope  of  T&E  Metadata,  no  single  standard  can  be  all  encompassing.  However, 
a  number  of  standards  exist  or  are  being  developed  that  cover  specific  sub-scopes  of  T&E 
metadata.  Some  of  these  standards  are  identified  below. 

a.  Telemetry  Attributes  Transfer  Standard  (TMATS).  A  well-known  text-based  fonnat 
maintained  by  the  RCC  Telemetry  Group  (TG)  for  describing  the  attributes  required 
to  process  telemetered  data  (Reference  d). 

b.  Instrumentation  Hardware  Abstraction  Language  (IHAL).  An  XML-based  language 
for  describing  the  capabilities  and  configuration  of  data  acquisition  instrumentation 
(Reference  e).  The  IHAL  is  currently  under  review  by  the  RCC  Telemetry  Group 
(TG)  for  inclusion  as  one  of  their  published  standards. 
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c.  Metadata  Description  Language  (MDL).  An  XML-based  language  for  describing 
measurements,  including  their  conversions,  data  formats,  and  transmission.  The 
MDL  language  was  developed  as  part  of  the  Central  T&E  Investment  Program 
(CTEIP)  iNet  program  (Reference  f). 

d.  T&E  Markup  Language  (TEML).  An  XML-based  language  developed  at  Aberdeen 
Proving  Ground  (APG)  for  configuring  the  instrumentation  systems  used  in  their 
tests.  They  have  approached  several  vendors  about  adding  native  support  for  TEML. 

e.  Test  and  Training  Enabling  Architecture  (TENA).  An  object  model,  software 
architecture,  and  toolset  to  enable  the  exchange  of  test  information  between 
organizations  (Reference  g).  While  TENA  is  not  a  standard  metadata  language  or 
format,  it  does  define  a  standard  terminology  to  use  when  exchanging  test  data,  and 
provides  standard  definitions  for  common  concepts  such  as  time. 

f.  Data  Dictionary.  A  data  dictionary  of  common  T&E  terms  maintained  by  the  Army 
T&E  Command  (ATEC).  The  purpose  of  this  data  dictionary  is  to  serve  as  a 
controlled  vocabulary  for  Army  T&E  operations.  This  dictionary  of  terms  and  their 
definitions  helps  to  disambiguate  overloaded  terms  and  ensure  shared  understanding 
of  T&E  terminology. 

g.  T&E  Metadata  Reference  Model.  A  high-level  T&E  Metadata  Reference  Model 
published  by  the  RCC  Data  Sciences  Group  (Reference  a).  The  purpose  of  this  model 
is  to: 

(1)  Capture,  at  a  global  level,  the  types  of  metadata  required  to  completely  describe  a 
test. 

(2)  Provide  a  common  terminology  that  can  be  used  when  sharing  data  and  metadata 
among  organizations. 

(3)  Serve  as  a  guideline  for  test  organizations  to  ensure  a  more  comprehensive 
capture  of  metadata. 

(4)  Enable  and  guide  the  development  of  T&E  metadata  management  systems  by 
providing  a  common  high-level  information  model. 

2.2  Metadata  Creation 

The  complete  recommended  list  of  types  of  metadata  that  should  be  captured  is 
documented  in  the  Test  &  Evaluation  Metadata  Reference  Model  (Reference  a).  Most  of  the 
items  in  this  model  are  already  commonly  captured  by  most  test  organizations.  However,  during 
our  studies  at  the  various  test  organizations,  we  identified  several  types  of  metadata  that  were 
being  captured  by  only  a  few  test  organizations,  but  that  proved  to  be  highly  useful  and  thus 
worthy  of  a  special  mention: 

a.  Test  Event  Log.  During  playback,  the  start  and  end  times  for  each  test  point  or  other 
important  event  (expected  or  unexpected)  are  recorded  electronically.  This  creates  a 
single  log  that  identifies  each  of  the  important  periods  for  which  the  data  should  be 
studied.  Capturing  this  log  in  one  place  during  playback  eliminates  the  need  to  go  back 
after-the-fact  and  cross-reference  multiple  logs. 
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b.  Test  Asset  Versions.  Test  metadata  should  include  a  description  of  the  versions  of  all  test 
assets  used  during  the  test,  including  versions  of  all  software  and  firmware  used  during 
the  test.  Capturing  this  information  is  critical  if  at  any  time  in  the  future  some  aspect  of 
the  test  needs  to  be  recreated.  Additionally,  it  enables  the  quick  identification  of  all  tests 
that  may  be  affected  if  a  software  defect  is  identified. 

c.  Policy  Conformance.  It  is  important  to  document  the  information  necessary  to 
demonstrate  that  relevant  test  organization  policies  are  being  followed  during  a  test.  This 
enables  the  test  organization  to  respond  adequately  to  audits  from  federal  or  other 
regulatory  agencies.  This  is  especially  relevant  to  environmental  policies,  as  there  is 
growing  concern  over  environmental  impacts. 

2.3  Metadata  Storage  and  Access 

Among  the  test  organizations  visited  for  this  task,  it  was  almost  universally 
acknowledged  that  some  form  of  central  metadata  catalog  system  (as  described  in  Reference  h) 
should  be  put  in  place  to  handle  the  storage  of  test  metadata.  The  metadata  loaded  into  such  a 
system  may  not  necessarily  conform  to  any  single  format.  However,  the  structure  imposed  by  a 
central  catalog  allows  (and  in  some  cases,  forces)  engineers  to  associate  metadata  with  the 
information  accessible  through  the  catalog.  For  example,  storing  reports  in  a  document 
management  system,  rather  than  a  simple  file  system,  enables  additional  pieces  of  information 
such  as  the  author,  revision,  and  test  date  to  be  easily  associated  with  the  report  in  a  standard 
way. 


A  metadata  catalog  should  be  logically  centralized  but  could  be  physically  distributed. 
That  is,  the  catalog  does  not  necessarily  require  that  all  metadata  be  stored  in  a  single  physical 
location.  There  are  cases  where  different  systems  and  storage  strategies  work  better  for  different 
types  of  metadata.  It  is  best  to  allow  users  to  create  metadata  with  the  most  appropriate  system 
for  the  job,  and  provide  a  central  catalog  that  can  access  metadata  regardless  of  its  location. 

Furthennore,  a  metadata  catalog  should  present  the  information  to  the  user  according  to 
that  user’s  role.  Because  different  users  may  require  access  in  different  orders  or  generation  of 
metadata  in  different  orders,  custom  views  make  their  jobs  more  efficient. 

That  said,  it  is  important  that  the  catalog  not  be  overly  constrained  to  the  ideal  process 
flow.  Specifically,  it  should  not  require  that  one  piece  of  information  be  added  before  another. 
Creating  such  constraints  can  render  the  system  unusable  if  metadata  documentation  is 
frequently  presented  in  an  inconsistent  order. 
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2.4  Data  and  Metadata  Archiving 

This  document  thus  far  has  dealt  mostly  with  current  test  data  and  metadata.  Equally 
important  is  that  test  organizations  develop,  document,  and  conform  to  data  and  metadata 
retention,  archival,  and  backup  policies.  Generally,  archived  test  information  is  stored 
differently  than  current  test  information.  A  well-documented  policy  should  describe: 

a.  Which  information  should  be  kept. 

b.  How  long  to  keep  the  information. 

c.  Which  formats  are  to  be  used. 

d.  The  format  and  media  refresh  policies;  that  is,  how  often  the  data  should  be 
transferred  to  a  new  medium  or  a  new  format. 

e.  The  security  restrictions  and  classification  of  the  data. 

Since  policies  may  change  over  time,  a  snapshot  of  the  current  archival  policy  should  be 
stored  with  the  archived  information,  so  that  users  will  know  under  which  policy  it  was  archived. 
In  this  sense,  the  archival  policy  itself  becomes  a  source  of  metadata  associated  with  the  test. 
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CHAPTER  3 

CONCLUSIONS 

Following  a  study  of  representative  test  organizations,  the  DSG  has  documented  a 
number  of  best  practices  on  how  RCC  member  ranges  can  best  deal  with  T&E  metadata.  The 
practices  presented  in  this  document  are  meant  to  encourage  member  ranges  to  consider  the  types 
of  metadata  they  document,  the  fonnats  and  tenninology  used  for  describing  metadata,  the 
systems  used  for  storage  and  retrieval  of  metadata,  and  the  policies  used  for  archival  of  metadata. 
This  document  provides  guidance  to  test  organizations  in  development  of  their  T&E  metadata 
management  strategies,  while  potentially  avoiding  problems  others  have  faced. 
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